Vehicle-trailer rearview vision system and method

ABSTRACT

A method and system for generating a combined rearview image of an area behind a vehicle that is towing a trailer, the method including: capturing a first image from a vehicle camera that is installed on the vehicle, the first image being comprised of an area behind the vehicle; capturing a second image from a trailer camera that is installed on the trailer, the second image being comprised of an area behind the trailer; determining an obstructed viewing region within the first image, the obstructed viewing region being a region of the first image in which the trailer resides; overlaying the second image on the first image and at least partially within the obstructed viewing region; and inserting graphics in the first image, in the second image, and/or in a patch region within the obstructed viewing region and between the second image and the first image.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. Ser.No. 15/889,470 filed on Feb. 6, 2018, the entire contents of which areincorporated herein by reference.

FIELD

The present invention relates generally to generating and displaying anarea behind a vehicle and trailer through the use of cameras installedon both the vehicle and the trailer.

BACKGROUND

Many vehicles can come equipped with some type of towing package, whichtypically includes a trailer hitch and a trailer connector. The trailerconnector provides electrical power from the vehicle to the trailer andpowers electrical components on the trailer, such as tail lights and/orelectric braking devices. Some trailers now include a camera that ispositioned on the rear end of the trailer and that faces an area behindthe trailer. The video feed that is captured by the trailer camera canbe communicated to the towing vehicle (i.e., the vehicle that is towingthe trailer) and displayed for viewing by a vehicle operator orpassenger. Additionally, many vehicles include a rear-facing camera thatfaces an area behind the vehicle. However, the field of view of therear-facing camera of the vehicle is oftentimes obstructed by thetrailer.

SUMMARY

According to one embodiment, there is provided a method of generating acombined rearview image of an area behind a vehicle that is towing atrailer, the method including: capturing a first image from a vehiclecamera that is installed on the vehicle, the first image being comprisedof an area behind the vehicle; capturing a second image from a trailercamera that is installed on the trailer, the second image beingcomprised of an area behind the trailer; determining an obstructedviewing region within the first image, the obstructed viewing regionbeing a region of the first image in which the trailer resides;overlaying the second image on the first image and at least partiallywithin the obstructed viewing region; and inserting graphics in thefirst image, in the second image, and/or in a patch region within theobstructed viewing region and between the second image and the firstimage.

According to various embodiments, this method may further include anyone of the following features or any technically-feasible combination ofthese features:

-   -   the step of determining an image overlay area within the first        image and/or at least partly overlapping the obstructed viewing        region, wherein the second image is overlaid within the image        overlay area;    -   the step of determining a trailer articulation angle, wherein        the image overlay area is determined based at least partly on        the trailer articulation angle;    -   the step of processing the second image so that the second image        after the processing depicts the area behind the trailer from a        perspective having a viewing angle that is the same as a viewing        angle of the vehicle camera;    -   the processing step is based at least partly on a trailer camera        viewing angle and a vehicle camera viewing angle;    -   at least some of the graphics do not depict or attempt to depict        the area behind the vehicle or the area behind the trailer, and        wherein the at least some of the graphics highlight one or more        roadway features;    -   at least some of the graphics depict or attempt to depict the        area behind the vehicle and/or the area behind the trailer;    -   the at least some of the graphics are obtained using scene        memorization techniques, wherein the scene memorization        techniques include the steps of: retaining at least a portion of        the first image and, subsequently, recalling the portion of the        first image, and using the recalled portion of the first image        to fill in at least a portion of the patch regions within the        obstructed viewing region; and/or    -   the step of displaying the combined rearview image on an        electronic display mirror included in the vehicle, wherein the        electronic display mirror is a rearview mirror that enables a        vehicle operator or a vehicle passenger to view an area behind        the vehicle, and wherein the display of the combined rearview        image enables the vehicle operator or the vehicle passenger to        view the area behind the vehicle and the area behind the trailer        without being obstructed by the trailer.

According to another embodiment, there is provided a method ofgenerating a combined rearview image of an area behind a vehicle that istowing a trailer, the method including: capturing a first image from avehicle camera that is installed on the vehicle, the first image beingcomprised of an area behind the vehicle; capturing a second image from atrailer camera that is installed on the trailer, the second image beingcomprised of an area behind the trailer; defining an obstructed viewingregion within the first image, the obstructed viewing region being aregion within the first image that includes the trailer; determining animage overlay area within the obstructed viewing region; inserting thesecond image on the first image according to the image overlay area;rendering graphics in a patch region within the obstructed viewingregion and outside the image overlay area, wherein the graphics includenatural scene graphics that depict or attempt to depict an area withinthe obstructed viewing region and outside the second image; anddisplaying a combined image on an electronic display that is installedin the vehicle, wherein the combined image includes at least part of thefirst image, at least part of the second image, and the renderedgraphics.

According to various embodiments, this method may further include anyone of the following features or any technically-feasible combination ofthese features:

-   -   the method is at least partly carried out by an onboard computer        installed in the vehicle, wherein the onboard computer carries        out the defining step, the determining step, the inserting step,        and the rendering step;    -   the step of receiving the first image and the second image at        the onboard computer, wherein the second image is received via a        vehicle-trailer interface;    -   a plurality of iterations of the method are carried out        including a first iteration and a second iteration, wherein one        or more steps of the first iteration are carried out after one        or more steps of the second iteration, wherein the first        iteration is completed before the second iteration;    -   a plurality of iterations of the method are carried out        including a first iteration and a second iteration, wherein the        first iteration includes retaining at least a portion of the        first image in memory of the onboard computer during the a first        iteration, and wherein the rendering step of the second        iteration includes using the retained portion of the first image        to render the graphics of the second iteration;    -   the second iteration further comprises the step of obtaining a        speed of the vehicle, and wherein the rendering step of the        second iteration is at least partially based on the obtained        speed of the vehicle; and/or    -   the step of overlaying graphics on the combined image that        direct or attempt to direct a vehicle operator's or a vehicle        passenger's attention to a roadway feature.

According to yet another embodiment, there is provided a vehiclerearview vision system for use with a vehicle, including: a vehiclecamera installed on the vehicle, wherein the vehicle camera isconfigured to capture an area behind the vehicle; an electronic displayinstalled on the vehicle; an onboard computer installed on the vehicle,wherein the onboard computer includes a processor and a non-transitorycomputer-readable memory, wherein the memory includes a computer programproduct that, when executed by the processor, causes the vehicle tocarry out a combined image generation process that includes: obtaining afirst image from the vehicle camera, the first image being comprised ofthe area behind the vehicle; obtaining a second image from a trailercamera that is installed on a trailer being towed by the vehicle, thesecond image being comprised of an area behind the trailer; determiningan obstructed viewing region within the first image, the obstructedviewing region being a region of the first image in which the trailerresides; overlaying the second image on the first image and at leastpartially within the obstructed viewing region; inserting graphics in apatch region within the obstructed viewing region and between the secondimage and the first image; and displaying a combined image on theelectronic display, wherein the combined image includes at least part ofthe first image, at least part of the second image, and the insertedgraphics.

According to various embodiments, this system may further include anyone of the following features or any technically-feasible combination ofthese features:

-   -   a vehicle wheel speed sensor, and wherein some of the graphics        are generated based on a vehicle wheel speed reading from the        vehicle wheel speed sensors;    -   the memory includes image recognition software that enables the        onboard computer to identify roadway features and, wherein the        image recognition software is used in conjunction with the        computer program product to render graphics for display in        regions of the combined image corresponding to the identified        roadway features; and/or    -   the computer program product, when executed by the processor,        further causes the vehicle to retain at least a portion of the        first image in memory of the onboard computer during a first        iteration of the combined image generation process, and wherein        the rendering step of a second iteration of the combined image        generation process includes using the retained portion of the        first image to render the graphics of the second iteration.

DRAWINGS

Preferred exemplary embodiments will hereinafter be described inconjunction with the appended drawings, wherein like designations denotelike elements, and wherein:

FIG. 1 is a block diagram depicting an example of systems that may beused with a vehicle and a trailer;

FIG. 2 is a top view of a vehicle and a trailer, where the articulationangle of the trailer is zero;

FIG. 3 is a top view of a vehicle and a trailer, where the articulationangle of the trailer is non-zero;

FIG. 4 is a schematic view of a display on an electronic display mirrorshowing a perspective of a vehicle camera when the trailer articulationangle is zero;

FIG. 5 is a schematic view of a display on an electronic display mirrorshowing the perspective of a vehicle camera when the trailerarticulation angle is non-zero;

FIG. 6 is a flowchart illustrating embodiments of a method of generatinga combined rearview image of an area behind a vehicle that is towing atrailer;

FIG. 7 is a view of an electronic display mirror showing the perspectiveof a vehicle camera facing an area behind a vehicle that is towing atrailer;

FIG. 8 is a view of an electronic display mirror showing the perspectiveof a vehicle camera facing an area behind a vehicle that is towing atrailer and that is supplemented by images or video from a trailercamera;

FIG. 9 is a view of an electronic display mirror showing the perspectiveof a vehicle camera facing an area behind a vehicle that is towing atrailer, that is supplemented by images or video from a trailer camera,and that includes lane marker graphics; and

FIG. 10 is an embodiment of a combined image that includes a view of anelectronic display mirror showing the perspective of a vehicle camerafacing an area behind a vehicle that is towing a trailer, that issupplemented by images or video from a trailer camera, and that includeslane marker graphics, as well as background graphics.

DESCRIPTION

The systems and method described herein are generally designed togenerate a combined rearview image of an area behind a vehicle that istowing a trailer. The combined rearview image is an image that includesat least part of a first image captured by a vehicle camera, at leastpart of a second image captured by a trailer camera, and one or moregraphics. It should be appreciated that while certain areas or regionsof the combined rearview image may still be obstructed by one or moreobjects, the combined rearview image described herein substantiallymitigates or reduces the amount of obstruction caused by the trailerthat the vehicle is towing. In one embodiment, a method can be used tocapture a first image from a vehicle camera and to capture a secondimage from a trailer camera. Thereafter, the method can determine anobstructed viewing region within the first image that the trailer isobstructing and, then, the second image can be inserted into anappropriate area within the obstructed viewing region area. Thereafter,portions of the obstructed viewing region that are not covered by thesecond image can be filled in by using scene memorization techniques,which will be described in more detail below. Additionally, graphics canbe generated and displayed over the image so as to augment the image.These graphics can highlight roadway features, such as lane markers,curbs, and the edges of the roadway.

In some embodiments, the method may take into account scenarios wherethe articulation angle of the trailer is nonzero such that thelongitudinal axes of the trailer and the vehicle are not aligned withone another. For example, the method can include determining anarticulation angle of the trailer with respect to the vehicle usingreadings from various vehicle sensors as well as image processingtechniques that are carried out on the image received from the vehiclecamera. Once the articulation angle is determined, the vehicle maydetermine where to place the image that was captured by the trailercamera within the first image that was captured by the vehicle.

Also, in some embodiments, graphics can be generated and displayed overthe combined image (i.e., the final image including the first image fromthe vehicle camera with the second image from the trailer camerasuperimposed over the first image) to show roadway features, includingcurbs, lane markers, and edges of the roadway. These graphics can beused to supplement the combined image and can be generated duringvarious embodiments of the method provided herein.

With reference to FIG. 1, there is shown a potential embodiment of avehicle rearview vision system 10 for use with a vehicle and a trailer.Vehicle rearview vision system 10 includes a vehicle 12, trailer 14, andone or more vehicle-trailer interfaces (or connectors) 60,62. Thevehicle and trailer are electrically coupled to one another via thevehicle-trailer interfaces (or connectors) 60,62, and are structurallyor mechanically coupled to another via a trailer hitch. The vehicleincludes vehicle hardware 20 and the trailer includes trailer hardware70.

Vehicle 12 is depicted as a passenger car, but it should be appreciatedthat the present method and system may be implemented with othervehicles including motorcycles, trucks (including semi-trucks), sportsutility vehicles (SUVs), recreational vehicles (RVs), marine vessels,aircrafts, trains, etc. Some components of vehicle hardware 20 that aremore relevant to the present systems and method are shown in FIG. 1,although skilled artisans will appreciate that a much more extensivecollection of vehicle hardware exists in most modern vehicles. Thevehicle hardware 20 may include a vehicle camera 22, an onboard computer30, a body control module (BCM) 40, an electronic display mirror 50, anda first vehicle-trailer interface connector 60. It should also beappreciated that the vehicle hardware 20 shown in FIG. 1 is only forpurposes of illustration, as the actual arrangement or configuration ofcomponents, devices, modules and/or systems could vary substantiallyfrom that shown here and it is not limited to any particular embodiment.

Vehicle camera 22 can be used to capture photographs, videos, and/orother information pertaining to light. Camera 22 can be an electronicdigital camera that is powered through use of a vehicle battery. Camera22 may include a memory device and a processing device to store and/orprocess data that it captures or otherwise obtains. The data obtained bythe camera 22 may be sent to another vehicle system module (VSM) such asthe on-board computer 30 of the vehicle. Camera 22 may be of anysuitable camera type (e.g., charge coupled device (CCD), complementarymetal oxide semiconductor (CMOS), etc.) and may have any suitable lensso that it is not limited to any particular type, brand, or model.According to one possibility, camera 22 provides video data where theimages are horizontally flipped (i.e., they are the mirror image of thevideo originally captured by the camera). The horizontally flipped videodata accounts for the fact that camera 22 and the driver are facingopposite directions thereby allowing presentation of the video data onthe electronic display mirror 50 in an orientation that is consistentwith a traditional rear view mirror. Some non-limiting examples ofpotential embodiments or features that may be used with camera 22include: infrared LEDs for night vision; wide angle or fish eye lenses;surface mount, flush mount, license mount, or side mount cameras;stereoscopic arrangements with multiple cameras; cameras integrated intotail lights, brake lights, or other components at the rear end of thevehicle; and wired or wireless cameras, to cite a few possibilities.

Camera 22 can be installed and/or mounted on vehicle 12 and may beconfigured to face in a rearward direction with respect to the vehiclesuch that the camera can capture an area located behind the vehicle—thatis, the field of view of the camera may include an area behind thevehicle and, in many embodiments, a trailer that is being towed byvehicle 12 may occupy at least a portion of the field of view of thecamera. According to a particular embodiment, camera 22 can be mountedon a rear exterior portion of vehicle 12 and, in some embodiments,camera 22 may be a backup camera (or reversing camera) that is alreadyincluded as a part of many consumer vehicles, including cars and trucks,or that may be required by one or more laws or regulations, includingthose regulations of the National Highway Traffic Safety Administration(NHTSA) that requires certain vehicles to include a backup camera. Inone embodiment, the camera 22 may be mounted on or embedded within arear bumper of vehicle 12, a trunk or other rear door of vehicle 12, atailgate (including those included in pickup trucks) of vehicle 12, aspoiler of vehicle 12, and/or any other location on vehicle 12 that issuitable for mounting or embedding camera 22 such that the field of viewincludes an area behind vehicle 12.

In some embodiments, multiple cameras 22 can be used, each of which canbe mounted and/or installed on vehicle 12. In one particular embodiment,multiple cameras can be positioned on the exterior of the vehicle andfacing in the rearward direction of the vehicle. Two or more cameras maybe configured in a stereoscopic orientation such that video data isprovided from multiple perspectives of an area and, when combined andprocessed according to a three-dimensional rendering algorithm, athree-dimensional reconstruction of the area (e.g., the area behind thetrailer) may be rendered. This rendering may then be displayed on avisual display, such as visual display 58 or electronic display mirror50. A stereoscopic orientation refers to an orientation of multiplecameras such that their fields of view overlap thereby allowing multipleperspectives of the area to which their respective fields of viewoverlap.

Onboard computer 30 is a vehicle system module that includes a processor34 and memory 36 and, in some embodiments, can also include a wirelesschipset or circuitry 32 that enables wireless communications to becarried out with, for example, a cellular carrier system or anotherlocal wireless device. Additionally, at least in some embodiments,onboard computer 30 can be an infotainment unit (e.g., infotainment headunit, in-car entertainment (ICE) unit, in-vehicle infotainment (IVI)), avehicle head unit, a center stack module (CSM), or vehicle navigationmodule. In some embodiments, onboard computer 30 may include one or morevehicle-user interfaces including pushbutton 52 and/or microphone 56, asdiscussed more below.

Processor 34 can be any type of device capable of processing electronicinstructions including microprocessors, microcontrollers, hostprocessors, controllers, vehicle communication processors, andapplication specific integrated circuits (ASICs). It can be a dedicatedprocessor used only for onboard computer 30 or can be shared with othervehicle systems. Processor 34 executes various types of digitally-storedinstructions, such as software or firmware programs stored in memory 36,which enable the computer 30 to provide a wide variety of services. Forinstance, processor 34 can execute programs or process data to carry outat least a part of the method discussed herein. Memory 36 may includeRAM, other temporary powered memory, any non-transitorycomputer-readable medium (e.g., EEPROM), or any other electroniccomputer medium that stores some or all of the software needed to carryout the various external device functions discussed herein.

As mentioned above, onboard computer 30 may include a wireless chipsetor circuitry 32 that can be used to carry out wireless communicationsusing antenna 38. In one embodiment, wireless chipset 32 is a cellularchipset that is configured to carry out cellular communications using acellular carrier system, such as those that may implement GSM/GPRStechnology, CDMA or CDMA2000 technology, LTE technology, etc.Additionally, or alternatively, onboard computer 30 may include ashort-range wireless communications (SRWC) circuit that enables SRWC tobe carried out using SRWC technologies including Wi-Fi™, WiMAX™,ZigBee™, Wi-Fi Direct™, other IEEE 802.11 protocol, Bluetooth™,Bluetooth™ Low Energy (BLE), or near field communication (NFC). In otherembodiments, wireless communications may be carried out at anothervehicle system module (VSM) that is connected to bus 44.

Body control module (BCM) 40 is shown in the exemplary embodiment ofFIG. 1 as being electrically coupled to communication bus 44. In someembodiments, the BCM 40 may be integrated with or part of a center stackmodule (CSM) and/or integrated with onboard computer 30. Or, the BCM maybe a separate device that is connected to one another via bus 44. BCM 40can include a processor and/or memory, which can be similar to processor34 and memory 36 of onboard computer 30, as discussed above. BCM 40 maycommunicate with onboard computer 30 and/or one or more vehicle systemmodules, such as audio system 54 or other VSMs 42. BCM 40 may include aprocessor and memory such that the BCM may direct one or more vehicleoperations including, for example, controlling central locking, airconditioning, power mirrors, controlling the vehicle primary mover(e.g., engine, primary propulsion system), and/or controlling variousother vehicle modules. BCM 40 may receive data from onboard computer 30and, subsequently, send the data to one or more vehicle modules.

Additionally, BCM 40 may provide information corresponding to thevehicle state or of certain vehicle components or systems. For example,the BCM may provide the onboard computer 30 with information indicatingwhether the vehicle's ignition is turned on, the gear the vehicle ispresently in (i.e. gear state), and/or other information regarding thevehicle. In one embodiment, the method 200 (FIG. 6) may be initiatedupon one or more vehicle states, such as a vehicle ignition on condition(i.e., a vehicle state where the ignition is turned on). And, in someembodiments, the method may use vehicle state information for carryingout one or more steps. In such embodiments, the BCM 40 may obtain andsend information to onboard computer 30, including vehicle speed,trailer articulation angle, steering wheel angle, and various otherinformation that may be obtained from other VSMs or sensors installed onthe vehicle. In some embodiments, a trailer articulation angle can bedetermined or measured using various vehicle sensors and, at least inone embodiment, the vehicle may include a sensor that is specificallyadapted or configured to measure the trailer articulation angle.

Electronic display mirror 50 includes a video display integrated intothe rear view mirror unit or housing so that the display shows thedriver the video output from the vehicle camera 22 and/or trailer camera72 in real time. According to the exemplary embodiment of FIG. 1, theelectronic display mirror 50 is communicatively coupled to and receivesimages and/or video from onboard computer 30. In one embodiment, theelectronic display mirror 50 may be directly wired to onboard computer30 or may be connected to onboard computer 30 via bus 44. The electronicdisplay mirror may utilize any number of different display technologies,such as a liquid crystal display (LCD) or a light emitting diode (LED)display. It should be appreciated that the system and method describedherein may use any suitable type of electronic display mirror 50 and arenot limited to any particular one. For instance, the electronic displaymirror 50 may be a full display mirror (FDM) where the entire viewablesurface of the device shows video from camera 22 and/or camera 72; itmay be the type of electronic display mirror where only a portion of thedisplay surface shows video from camera 22 and/or camera 72, whereasother portions of the display surface show other information (e.g.,turn-by-turn or other navigation instructions, a compass or headinginformation, host vehicle performance data, etc.); it could have daytimeand nighttime modes that adjust the brightness accordingly; or it couldhave a touchscreen so that users can make selections or otherwise inputdata, to cite a few of the possibilities. Instead of the electronicdisplay mirror 50 being positioned in the traditional location of a rearview mirror (i.e., mounted to the front windshield at an upper, centerlocation), it is possible for it to be part of onboard computer 30,other center console infotainment unit, or other VSM.

Vehicle hardware 20 also includes a number of vehicle user interfacesthat provide vehicle occupants with a means of providing and/orreceiving information, including pushbutton(s) 52, audio system 54,microphone 56, and visual display 58. As used herein, the term“vehicle-user interface” broadly includes any suitable form ofelectronic device, including both hardware and software components,which is located on the vehicle and enables a vehicle user tocommunicate with or through a component of the vehicle. Thepushbutton(s) 52 allow manual user input into the onboard computer 30 toprovide other data, response, or control input. Audio system 54 providesaudio output to a vehicle occupant and can be a dedicated, stand-alonesystem or part of the primary vehicle audio system. According to theparticular embodiment shown here, audio system 54 is operatively coupledto both vehicle bus 44 and an entertainment bus (not shown) and canprovide AM, FM and satellite radio, CD, DVD and other multimediafunctionality. This functionality can be provided in conjunction with orindependent of an infotainment module and/or onboard computer 30.Microphone 56 provides audio input to the onboard computer 30 to enablethe driver or other occupant to provide voice commands and/or carry outhands-free calling via a wireless carrier system. For this purpose, itcan be connected to an on-board automated voice processing unitutilizing human-machine interface (HMI) technology. Visual display ortouch screen 58 is preferably a graphics display, such as a touch screenon the instrument panel or a heads-up display reflected off of thewindshield, and can be used to provide a multitude of input and outputfunctions. Various other vehicle user interfaces can also be utilized,as the interfaces of FIG. 1 are only an example of one particularimplementation.

Any of the devices 22-62 may be stand-alone, as illustrated in FIG. 1,or they may be incorporated or included within some other device, unitor module. Furthermore, any of the devices 22-62 may be dedicated orthey may be part of or shared by other systems or sub-systems in thevehicle. For example, the enhanced video output from onboard computer 30may be directly or indirectly provided to the electronic display mirror34. Accordingly, the devices 22-62 are not limited to the schematicrepresentation in FIG. 1 or the exemplary descriptions above, nor arethey limited to any particular embodiment or arrangement so long as theycan be used with the method described herein.

Trailer 14 is depicted as a storage trailer, but it should beappreciated that the present systems and method may be implemented withother trailers including boat trailers, livestock trailers,semi-trailers, motorcycle trailers, popup camper trailers, pushertrailers, travel trailers, fifth wheels, etc. Some components of trailerhardware 70 that are more relevant to the present systems and method areshown in FIG. 1, although skilled artisans will appreciate that a muchmore extensive collection of trailer hardware exists in modern trailers.Trailer 14 can include a frame 80 that provides structural support, aswell as certain trailer frame components 82,84 that can be used byvehicle 12 to determine a trailer articulation angle, as discussed morebelow. The trailer 14 may also include one or more cameras 72 and avehicle-trailer interface 62. It should also be appreciated that thetrailer hardware 70 shown in FIG. 1 is only for purposes ofillustration, as the actual arrangement or configuration of components,devices, modules and/or systems could vary substantially from that shownhere and it is not limited to any particular embodiment. In anotherembodiment, a second trailer may be attached to trailer 14 via aninterface similar to the vehicle-trailer interface such that the vehiclemay communicate with the second trailer via a trailer-trailer interface(which may be similar or the same in construction as the vehicle-trailerinterface). Additional trailers may be attached and communicated with bythe vehicle or other trailers in a like arrangement.

Trailer camera 72 can be used to capture photographs, videos, and/orother information pertaining to light. Camera 72 can be an electronicdigital camera that is powered through use of a vehicle battery. Camera72 may include a memory device and a processing device to store and/orprocess data that it captures or otherwise obtains. The data obtained bythe camera 72 may be sent to a module located on vehicle 12 such ason-board computer 30. Camera 72 may be of any suitable camera type(e.g., charge coupled device (CCD), complementary metal oxidesemiconductor (CMOS), etc.) and may have any suitable lens so that it isnot limited to any particular type, brand, or model. According to onepossibility, camera 72 provides video data where the images arehorizontally flipped (i.e., they are the mirror image of the videooriginally captured by the camera). Some non-limiting examples ofpotential embodiments or features that may be used with camera 72include: infrared LEDs for night vision; wide angle or fish eye lenses;surface mount, flush mount, license mount, or side mount cameras;stereoscopic arrangements with multiple cameras; cameras integrated intotail lights, brake lights, or other components at the rear end of thetrailer; and wired or wireless cameras, to cite a few possibilities.

Camera 72 can be installed and/or mounted on trailer 14 and may beconfigured to face in a rearward direction with respect to the trailersuch that the trailer camera can capture an area located behind thetrailer—that is, the field of view of the trailer camera may include anarea behind the trailer and, in many embodiments, the images or videocaptured by trailer camera 72 can be used to supplement images or videocaptured by vehicle camera 22. In one embodiment, the images or videocaptured by trailer camera 72 can be incorporated into images or videocaptured by vehicle camera 22 in an obstructed viewing region. Forexample, trailer 14 may obstruct a region of the image captured byvehicle camera 22 (i.e., the obstructed viewing region) and, thus, thisobstructed region can be supplemented by combining images or video fromtrailer camera 72 into the obstructed viewing region contained withinthose images or video from vehicle camera 22. In this way, images orvideo from vehicle camera 22 and trailer camera 72 can be used togetherand integrated with one another so as to provide an unobstructed rearview that can be displayed on electronic display mirror 50 for viewingby a vehicle operator or passenger.

According to a particular embodiment, trailer camera 72 can be mountedon a rear exterior portion of trailer 14 and, in some embodiments,trailer camera 72 may be a backup camera (or reversing camera) that isalready included as a part of some consumer trailers. In one embodiment,the camera 72 may be mounted on or embedded within a rear bumper oftrailer 14, a trunk or other rear door of trailer 14, and/or any otherlocation on trailer 14 that is suitable for mounting or embedding camera72 such that the field of view includes an area behind trailer 14. Inother embodiments, camera 72 may be mounted or installed on a side oftrailer 14 and facing a blind spot area that cannot easily be viewed bya vehicle operator or which may be mounted or installed on a location oftrailer 14 such that the field of view of camera 72 captures images orvideo of an area constituting at least a portion of the obstructedviewing region as discussed above. In some embodiments, multiple cameras72 can be used, each of which can be mounted and/or installed on trailer14. In one particular embodiment, multiple cameras can each bepositioned on opposite sides of the exterior of the trailer so as toface one or more blind spot areas (i.e., areas in which a vehicleoperator cannot easily view). In other embodiments, the multiple camerasmay be mounted and/or installed on trailer 14 such that the cameras facein the rearward direction of the trailer. Two or more cameras may beconfigured in a stereoscopic orientation such that as that which isdiscussed above with respect to vehicle camera 22.

Vehicle-trailer interface connectors 60 (vehicle-side) and 62(trailer-side) may be used to connect the trailer hardware 70 toelectric power provided by the vehicle and/or to provide communicationsbetween trailer hardware 70 and vehicle hardware 20. In one embodiment,a standard 7-pin SAE J560 compliant interface can be used to connect thetrailer hardware 70 to power from a vehicle battery. In manyembodiments, the vehicle-trailer interface can include a dedicated linefor communications and, in other embodiments, trailer hardware 70 mayinclude wireless circuitry that can be used to carry out wirelesscommunications with onboard computer 30 via use of wireless chipset 32.In either wireline or wireless embodiments, the vehicle and trailer maycommunicate in a bidirectional manner. Such wireless communications mayinclude any of those SRWC technologies discussed above, such as any ofthe IEEE 802.11 protocols or any of the Bluetooth™ protocols. In otherembodiments, power line communication techniques can be used to modulatedata over one or more power lines connecting vehicle 12 and trailer 14.Those skilled in the art will appreciate other potential vehicle-trailercommunication means that may be integrated into vehicle hardware 20and/or trailer hardware 70.

The vehicle-trailer interface can include two connectors, a vehicle-sideconnector 60 and a trailer-side connector 62 that fit together in acomplementary fashion. In one embodiment, vehicle-side interfaceconnector 60 which is included in vehicle hardware 20 includes one ormore female and/or male pins. Accordingly, trailer-side interfaceconnector 62 which is included in trailer hardware 70 includescomplementary male and/or female pins such that they complementvehicle-side interface connector 60 pins.

With reference to FIGS. 2 and 3, there is provided top views of avehicle and a trailer, where the articulation angle of the trailer iszero (FIG. 2) and nonzero (FIG. 3). FIGS. 2 and 3 can be used as areference to supplement the discussion of method 200 (FIG. 6), which isdiscussed below. As illustrated in FIGS. 2 and 3, reference line Aillustrates a longitudinal central axis of vehicle 12, reference line Billustrates a longitudinal central axis of trailer 14, and referenceline C illustrates an axis of a frame component 82,84 of trailer 14. Asdiscussed more below, certain angles may be calculated using thesereference points to provide a trailer articulation angle that can beused in various embodiments of method 200 (FIG. 6).

As shown in FIG. 2, there is shown a scenario where vehicle 12 andtrailer 14 are aligned with one another such that the presentarticulation angle ϕ of the trailer with respect to the vehicle is zero.The articulation angle can be measured from reference line A andreference line B such that the articulation angle represents an anglebetween the central longitudinal axis of vehicle 12 and the centrallongitudinal axis of trailer 14. In some embodiments, other referencelines or axes can be used to calculate the articulation angle ϕ. Forexample, the vehicle can use rear-facing vehicle camera 22 to determinethe articulation angle ϕ based on one or more objects within the fieldof view of camera 22, including a trailer frame component 82 or 84. Asshown in FIG. 2, reference line C, which corresponds to an axis oftrailer frame component 82, and reference line A are used to calculate afirst reference angle α₁ that can be used with a predetermined orpredefined reference angle α₀ to determine an articulation angle ϕ. Forexample, vehicle camera 22 can be used to capture images of trailerframe component 82 and, thereafter, can use image processing techniquesto determine a first reference angle α₁. This angle can then be comparedto a predetermined or predefined reference angle α₀, which can be anangle between reference line C and reference line A when thelongitudinal vehicle axis (represented by reference line A) and thelongitudinal trailer axis (represented by reference line B) are aligned,as depicted in FIG. 2. In some embodiments, the first reference angle α₁can be subtracted from the predetermined reference angle α₀ to obtainthe articulation angle ϕ. The articulation angle ϕ can be used invarious steps of method 200, which is discussed below.

With reference to FIGS. 4 and 5, there is shown examples of a view takenfrom vehicle camera 22 during the scenarios presented in FIG. 2 and FIG.3, respectively. These views illustrate a first image 102, a secondimage 104, an obstructed viewing region 110, and an image overlay area120. FIG. 4 shows trailer 14 straight on as would be captured by vehiclecamera 22 in a scenario where the trailer articulation angle ϕ is 0°(FIG. 2) and FIG. 5 shows trailer 14 at an angle as would be captured byvehicle camera 22 in a scenario where the trailer articulation angle ϕis non-zero (FIG. 3). When describing method 200 (FIG. 6) below,reference to FIGS. 2-5 is made for purposes of illustration.

With reference to FIG. 6, there is provided a method 200 of generating acombined rearview image of an area behind a vehicle that is towing atrailer. The method 200 generally includes the steps of capturing images(or video) from vehicle camera 22 and trailer camera 72, determining anobstructed viewing region with the images (or video) from vehicle camera22, superimposing the images (or video) from trailer camera 72 over atleast part of the obstructed viewing region, and applying scenememorization techniques to fill the area of the obstructed viewingregion that was not supplemented by the images (or video) from trailercamera 72. Other embodiments include augmenting the finished or combinedrearview image (or video) with graphics that, for example, can highlightroadway features.

In many embodiments, method 200 can be implemented in a computer programproduct (or “application”) embodied in a computer readable medium andincluding instructions usable by one or more processors of one or morecomputers of one or more systems. The computer program product mayinclude one or more software programs comprised of program instructionsin source code, object code, executable code or other formats; one ormore firmware programs; or hardware description language (HDL) files;and any program related data. The data may include data structures,look-up tables, or data in any other suitable format. The programinstructions may include program modules, routines, programs, objects,components, and/or the like. The computer program can be executed on onecomputer or on multiple computers in communication with one another.

The program(s) can be embodied on computer readable media, such asmemory 36, which can be non-transitory and can include one or morestorage devices, articles of manufacture, or the like. Exemplarycomputer readable media include computer system memory, e.g. RAM (randomaccess memory), ROM (read only memory); semiconductor memory, e.g. EPROM(erasable, programmable ROM), EEPROM (electrically erasable,programmable ROM), flash memory; magnetic or optical disks or tapes;and/or the like. The computer readable medium may also include computerto computer connections, for example, when data is transferred orprovided over a network or another communications connection (eitherwired, wireless, or a combination thereof). Any combination(s) of theabove examples is also included within the scope of thecomputer-readable media. It is therefore to be understood that themethod can be at least partially performed by any electronic articlesand/or devices capable of carrying out instructions corresponding to oneor more steps of the disclosed method, including the electronic devicesthat are a part of the vehicle hardware 20 and/or the trailer hardware70.

Method 200 begins with step 210, wherein a first image from a vehiclecamera that is installed on the vehicle is captured. In manyembodiments, the first image can be comprised of an area behind thevehicle. And, in other embodiments, multiple vehicle cameras can beused, such as camera facing a left blind spot and another camera facinga right blind spot. The images or video captured by camera 22 can thenbe sent to onboard computer 30 via communications bus 44. The onboardcomputer 30 obtains this image/video input and then may, for example,store, buffer, process and/or pass the video to the electronic displaymirror 50, as will be described below. The method 200 continues to step220.

In step 220, a second image from a trailer camera that is installed onthe trailer is captured. In some embodiments, the second image can becomprised of an area behind the trailer, such as an area within theobstructed viewing region of the first image. The second image can becaptured by trailer camera 72 and then communicated to onboard computer30 of vehicle 12. The images can be sent to onboard computer via a wiredconnection that runs from the trailer to the vehicle via thevehicle-trailer interface connectors 60,62. In other embodiments, theimages or video from trailer camera 72 can be sent wirelessly using aSRWC chipset or circuitry at the trailer and vehicle. The onboardcomputer 30 obtains the second image input and then may, for example,store, buffer, process and/or pass the video to the electronic displaymirror 50, as will be described below. The method 200 continues to step230.

In step 230, the second image is processed and/or manipulated such thatthe second image reflects a viewing angle that corresponds to the firstimage. For example, in some embodiments, the vehicle camera 22 and thetrailer camera 72 may be oriented at different angles with respect to anarea behind the vehicle or trailer and, thus, prior to incorporating thesecond image into a region of the first image, it may, at least in someembodiments, prove beneficial to determine a viewing angle thatcorresponds to the first image so that the second image can be processedand/or manipulated in a manner so as to depict an area behind thetrailer according to the determined viewing angle. The viewing angle canbe determined using various methods, including using predetermined orpredefined viewing angle values that are associated with the vehiclecamera 22 and the trailer camera 72. In other embodiments, the firstimage can be processed and/or manipulated so that the first imagereflects a viewing angle that corresponds to the second image. Moreover,in at least one embodiment, both the first and the second image can beprocessed and/or manipulated so that the resultant image reflects aviewing image from the perspective of an angle of interest.

For example, the vehicle may be provisioned or programmed withinformation concerning the viewing angle of the vehicle camera 22.Additionally, the vehicle may obtain the viewing angle of the trailercamera 72 by, for example, retrieving trailer information concerning themodel of trailer 14 from a trailer information database. The trailerinformation database can be stored at a remote location, such as aremote server that is accessible over the Internet. In such a case,vehicle 12 can retrieve the trailer information (that includesinformation pertaining to the trailer camera viewing angle) and can thenstore this information in memory, such as memory 34. In otherembodiments, certain trailer information may be known to trailer 14 andsent to onboard computer 30 of vehicle 12. Or, in other embodiments,vehicle 12 may include a trailer information database, such as in memory36. Other examples of determining or obtaining the angle of the trailercamera can be by means of the driver entering that information to thesystem, or the system automatically learning the angle of the camerabased on an algorithm where the driver is instructed to drive for ashort period of a time while the camera learns its position. And, inanother embodiment, the system can determine this angle by prompting anoperator to place certain pre-defined shapes in certain locations in thefield of view (FOV) of the camera and, then, prompting the camera tolearn its location and viewing angle based on the location of thepre-defined shapes. Once the trailer camera viewing angle is obtained,the vehicle can also obtain the vehicle camera viewing angle. This anglecan be obtained from a vehicle information database that storesinformation pertaining to the vehicle. The onboard computer 30 canobtain this information from a remote server, from memory 36, and/orfrom another VSM of vehicle 12.

Once the trailer camera viewing angle and the vehicle camera viewingangle are obtained at onboard computer 30, the second image (from thetrailer camera 72) can be processed or manipulated accordingly. Forexample, certain areas of the second image may need to be distorted soas to reflect an image that corresponds to the viewing angle of thevehicle camera 22. Various techniques can be used to distort ortransform the second image. And, in some embodiments, the first imagecan be processed according to the viewing angles as well. As mentionedabove, images corresponding to other viewing angles may be used and, insuch embodiments, the method can be carried out accordingly as will beappreciated by those skilled in the art. Once the second image isprocessed according to the trailer camera viewing angle and/or thevehicle camera viewing angle, then the method 200 proceeds to step 240.

In step 240, an obstructed viewing region within the first image isdetermined. In many embodiments, the obstructed viewing region is aregion of the first image in which the trailer resides or occupies. Forexample, in FIG. 4, the trailer 14 is obstructing at least part ofregion 110 and, in FIG. 5, the trailer 14, which is articulated at anangle ϕ, is obstructing much of the left side of the first image 102.The obstructed viewing region can thus be dynamically determined basedon various factors including the articulation angle of the trailer 14,the steering wheel angle of vehicle 12, the identification of thetrailer 14 within the first image using image processing techniques, andvarious other information. Moreover, the obstructed viewing region caninclude other areas of the first image that are not obstructed and/orthat surround a region of the first image in which the trailer resides.In this way, these areas surrounding the trailer 14 within the firstimage can be removed so as to allow a smoother or more naturaltransition between the first and second image, which is inserted intothe first image, as discussed below (step 250).

The obstructed viewing region can be defined by two or more coordinatesthat correspond to locations within the first image. The coordinatesystem can be based on the resolution of the first image and can bedefined using indices (coordinate pairs) corresponding to a pixel withinthe first image. In one embodiment, the obstructed viewing region can bedefined by four coordinates that represent vertices of a quadrangle(e.g., a rectangle). In other embodiments, other processes can be usedto define the obstructed viewing region within the first image.

The obstructed viewing region can be statically defined or dynamicallydefined. In one embodiment, memory 36 of computer 30 can store apredefined obstructed viewing region definition (e.g., four coordinatesdefining the obstructed viewing region) that corresponds to an area thatis obstructed by the trailer when the articulation angle ϕ is 0° (zerodegrees) and the vehicle and trailer are on a leveled surface. Thepredefined obstructed viewing region definition can then be modified inlight of a present articulation angle ϕ of the trailer, image processingtechniques performed on the first image and/or the second image, thedistance between the trailer and the vehicle, a steering wheel angle,one or more other vehicle or trailer states as determined by vehiclehardware 20 or trailer hardware 70, and/or a variety of otherinformation (e.g., trailer width and height). The vehicle can obtainthis information, such as the distance between the trailer and thevehicle, by downloading this content from a remote facility or by beingpreconfigured or programmed with the information, which can be stored inmemory 36. Or, the vehicle can use backup sensors to obtain a distancebetween the trailer and the vehicle.

The articulation angle can be used to adjust the obstructed viewingregion definition since, depending on the articulation angle ϕ, thetrailer may obstruct different regions of the first image. And, in manyembodiments, the length of the trailer can be used to adjust theobstructed viewing region definition, as that can play a significantrole in ascertaining the boundaries of the obstructed viewing region.For example, if the trailer is articulated to the right (reference angleα₁ is less than the predetermined reference angle α₀), then the leftside of the first image from the vehicle camera 22 may be obstructedmore than the right side of the first image, as shown in FIG. 5. Thus,by obtaining the articulation angle ϕ, the onboard computer may knowwhether and to what extent certain portions of the first image are beingobstructed by the trailer thereby providing the onboard computer withinformation so as to define a suitable obstructed viewing region. Thearticulation angle ϕ can be obtained using various methods, such asusing image processing techniques on the first image to determine areference line C corresponding to a trailer frame component 82, 84and/or using information from BCM 40, including a steering wheel angle.

Additionally, when defining the obstructed viewing region, informationregarding the obstructed area and/or other areas that are to be cut outcan be derived from using image processing techniques on the first imageand/or the second image. For example, the trailer may be identified byprocessing the first image using object recognition or patternrecognition techniques, some of which may include the use of predefinedor predetermined data that is associated with the trailer 14. Theseimage recognition techniques can be used instead of the articulationangle, or may be used to corroborate one another in defining theobstructed viewing region. Additionally, or alternatively, the vehiclecould use already existing proximity sensors in back of the vehicle(i.e., backup sensors) that can use radar (or other techniques) tolocate the boundaries of the trailer.

Moreover, the first image and the second image may include regions thatrepresent the same area behind the trailer. For example, both thevehicle camera 22 and the trailer camera 72 may capture an objectlocated to the back-left of the trailer. Thus, in at least someembodiments, it may not be desirable to place the second image withinthe first image (step 250) wherein both images depict the same object,as this may confuse an operator or passenger of vehicle 12. Thus, imageprocessing techniques (including object recognition techniques) can beused to identify certain objects within the field of view of the vehiclecamera 22 and the trailer camera 72. These duplicate regions can then beidentified within the first image and can then be included in theobstructed viewing region definition, which will result in these regionsbeing removed from the first image before being displayed for viewing bya vehicle operator or passenger. Once the obstructed viewing regiondefinition is determined and stored in memory 36, the method 200proceeds to step 250.

In step 250, the second image is overlaid on the first image and withinthe obstructed viewing region. This step can include determining animage overlay area that corresponds to an area in which the second imagewill be overlaid on the first image. The image overlay area can bedefined using a plurality of coordinates similar to the manner in whichthe obstructed viewing region is defined. And, in some embodiments, theimage overlay area can be the same as the obstructed viewing region—thatis, the image overlay area definition can be the same as the obstructedviewing region definition. And, in other embodiments, the image overlayarea can be based on the obstructed viewing region definition, or theobstructed viewing region can be based on the image overlay area. And,in many embodiments, the obstructed viewing region may include a portionof the image overlay area and other portion(s) that are not within theimage overlay area. These portions of the obstructed viewing region thatare outside the image overlay area can be referred to as patch regionsand will likely be filled in, as described below.

The image overlay area can be determined using a variety of techniques,some of which can be based on the articulation angle ϕ of the trailer,trailer information (e.g., trailer dimensions such as trailer length),and other information. Object recognition or pattern recognitiontechniques can be used to align the second image within the first imageso as to not create a distorted or inaccurate appearance of objectswithin the final, combined rearview image. Once the image overlay areais determined, the definition of this area can be stored in memory, suchas memory 36, and may be stored along with other information, such asarticulation angle ϕ, so that, upon the later occurrence (as a part of afuture iteration of method 200) of step 250 during a scenario where thearticulation angle ϕ is the same, the image overlay area definition canbe recalled from memory 36 as opposed to recalculated. Thereafter, thesecond image data (e.g., the bytes representing the pixels) can betransposed into the first image data so that an image containing thefirst image and the second image within the image overlay area isobtained. This image (i.e., the “aggregate image”) may then be stored inmemory 36. Once the second image is overlaid or inserted into the imageoverlay position, it could prove useful to shift the second image aroundand/or adjust its size so that it fills the vacant space in a morelife-like way and better integrates the images together. By improvingthis integration process, there may be less patch region areas that needto be filled in. And, during this integration process, the second imagecan be resized, scaled, transformed, and/or processed using other imageprocessing techniques so as to integrate the second image into theaggregate image thereby forming a more life-like image. The method 200continues to step 260.

In step 260, graphics can be added so as to augment the image that is tobe displayed to a vehicle operator or passenger. These graphics canhighlight or direct a user's attention to certain objects or areaswithin the image thereby providing the vehicle operator or passengermore awareness of their surroundings. The graphics can correspond tocertain objects in the image, which can be identified by using certainimage processing techniques (object recognition or pattern recognitiontechniques). Alternatively or additionally, the graphics can be based onother information received at one or more vehicle sensors at thevehicle, such as information obtained by a lidar or radar, orinformation obtained from a wireless communications device via aconnection with a remote server. For example, the graphics can be a lanechange warning that warns a vehicle operator that an object, such asanother vehicle, is in a lane adjacent to the vehicle. The graphics canbe sized and/or transformed according to the corresponding object towhich they are to be applied to. In some embodiments, graphicshighlighting lane marker, curbs, and/or edges of the road can begenerated and displayed over the image (step 280).

Additionally, the onboard computer can obtain and/or generate graphicsfor the trailer wheels (i.e., virtual trailer wheel graphics) that canbe used to inform a vehicle operator or passenger of where the trailerwheels are relative to the road. The method 200 continues to step 270.

In step 270, graphics can be developed or obtained for use in filling inthe patch regions of the aggregate image. A variety of techniques can beused to fill these patch regions, including scene memorizationtechniques and/or image stitching techniques. In some embodiments,stitching techniques can be used to improve the overall image especiallyin regions included in or around the image overlay area and/or theobstructed viewing region. These techniques can include adjusting imageattributes (e.g., brightness, luminosity, contrast, color curves) aswell as selecting to alter individual pixel values.

The patch regions can also be filled through use of scene memorizationtechniques. Scene memorization refers to retaining certain images foruse later in other images that include at least a portion of the arearepresented by the retained images. For example, the vehicle camera 22may have an unobstructed view of areas to the sides of trailer 72. Then,as the vehicle moves forward, these areas may then become obstructed bytrailer 14. However, at this point, the vehicle can use the retainedimages to fill in the patch regions that are obstructed by the traileror that are within the obstructed viewing region—it should beappreciated that the obstructed viewing region may include areas of thefirst image that are not obstructed by the trailer but that are includedso that they will be removed and then patched or stitched therebyproviding a better transition between the first and second image.

Scene memorization techniques may be carried out by processor 34 ofonboard computer 30. The onboard computer 30 can use various informationfrom one or more vehicle system modules or sensors included on thevehicle, such as BCM 40 and/or a wheel speed sensor. In someembodiments, a vehicle speed may be obtained by the onboard computer 30from BCM 40 or from one or more wheel speed sensors installed on thevehicle 12. The vehicle speed can be used to determine a rate at whichthe pixels of the first and second image are moving and, thus, where toinsert the retained images within the obstructed viewing region. Themethod 200 continues to step 280.

In step 280, a combined image is displayed on an electronic displaymirror included in the vehicle. In other embodiments, the combined imagecan be displayed on another display of the vehicle, such as aninfotainment unit display or multimedia display. The combined imageincludes at least a part of the first image captured by the vehiclecamera 22, at least part of the second image captured by the trailercamera 72, and one or more graphics. The one or more graphics can begenerated to highlight roadway features or to otherwise direct a vehicleoperator's or passenger's attention to an object or other informationcontained within the combined image. Or, the one or more graphics can begraphics that attempt to recreate portions of the area surrounding thevehicle and trailer so as to imitate a view behind the vehicle that isnot obstructed by the trailer.

In many embodiments, the combined image can be displayed on electronicdisplay mirror 50. The combined image can be sent to the electronicdisplay mirror 50 via communications bus 44. In other embodiments, theunobstructed final image can be sent to one or more other displayswithin the vehicle, including display 58. The combined image, or atleast portions thereof, can be saved into a non-transitorycomputer-readable memory (or even a random access memory (RAM)) so thatthese images may be later recalled for use in carrying out futureiterations of the method 200. For example, upon a next iteration ofmethod 200, certain portions of the first image that were not within theobstructed viewing region may be recalled from memory and then used inthe next iteration to fill a portion of the obstructed viewing region.

Additionally, it should be appreciated that, although method 200 wasdiscussed with reference to creating a single image, method 200 can beused to create a series of images that can be used to create a video tobe displayed on electronic display mirror 50. And, at least in someembodiments, iterations of method 200 may be carried out in a serial orparallel manner—that is, method 200 may be completely carried out beforebeginning a next iteration of method 200 (serial) or method 200 (orcertain steps therein) may be carried out in a parallel manner such thatcertain steps of a first iteration of method 200 are carried out at thesame time as certain steps of a second iteration of method 200. Themethod 200 then ends.

With reference to FIGS. 7-10, there is shown a progression of videoand/or image data that is obtained using the vehicle camera and thetrailer camera, and that is combined with various graphics to obtain acombined image that depicts an area behind the vehicle while mitigatingthe interference caused by the trailer. In many embodiments, the method200 described above (FIG. 6) can be used to generate the combined image(FIG. 10) and then present the combined image on a display, such aselectronic display mirror 50.

FIG. 7 depicts a first image 102 that is obtained by a rear-facingvehicle camera 22 that shows trailer 14. As illustrated, the trailer 14,which is being towed by vehicle 12, obstructs a portion of the areabehind the trailer. This first image 102 can be captured by using thevehicle camera 22 as described in step 210 of method 200 (FIG. 6). FIG.8 depicts the first image 102, along with a second image 104 that iscaptured by a trailer camera 72, which is overlaid onto the first image102 at an image overlay area 120. FIG. 8 includes a patch portion 130that includes an area of the trailer 14 that is not supplemented by thesecond image 104 from the trailer camera 72. The second image 104 can becaptured by using the trailer camera 72 as described in step 220 ofmethod 200. Moreover, the second image 104 can be overlaid, repositionedand/or resized on the first image 102 at an image overlay position 120using a variety of different techniques, such as those described abovein step 250 of method 200.

FIG. 9 depicts the aggregate image from FIG. 8, but also includesvarious graphics that highlight one or more roadway features, such aslane marker graphics 140 and road edge graphic 150. These graphics canbe generated and/or rendered according to any of the embodimentsdiscussed above in step 260 of method 200 (FIG. 6). FIG. 10 depicts acombined image that is ready for presentation on a display of thevehicle, such as electronic display mirror 50. FIG. 10 depicts graphicsthat are generated so as to fill in the patch region 130 and, asillustrated, these graphics can be generated so that they match abackground scene behind the vehicle 12 and trailer 14. Scenememorization techniques, such as those discussed above in step 270, canbe used to fill in the patch region 130. Additionally, other graphicscan be added, such as trailer tire graphics 160 that depict a region inwhich the trailer wheels are located.

Those skilled in the art will appreciate that the system and/or methoddiscussed herein can be applied to other scenarios as well. In otherembodiments, a first camera can be included on and used by the hostvehicle (e.g., vehicle 12) and a second camera can be included onanother subject, such as another item being towed that is on a trailer(e.g., another vehicle, boat). Or, in other embodiments, a first camerathat is on the front of the host vehicle and that faces an area in frontof the host vehicle can be used with a second camera on the front ofanother vehicle that is in front of the host vehicle so as to enable thehost vehicle to obtain a combined image of an area in front of thevehicle and that is not (or less) obstructed by the vehicle in front ofthe host vehicle.

It is to be understood that the foregoing description is not adefinition of the invention, but is a description of one or morepreferred exemplary embodiments of the invention. The invention is notlimited to the particular embodiment(s) disclosed herein, but rather isdefined solely by the claims below. Furthermore, the statementscontained in the foregoing description relate to particular embodimentsand are not to be construed as limitations on the scope of the inventionor on the definition of terms used in the claims, except where a term orphrase is expressly defined above. Various other embodiments and variouschanges and modifications to the disclosed embodiment(s) will becomeapparent to those skilled in the art. For example, the specificcombination and order of steps is just one possibility, as the presentmethod may include a combination of steps that has fewer, greater ordifferent steps than that shown here. All such other embodiments,changes, and modifications are intended to come within the scope of theappended claims.

As used in this specification and claims, the terms “for example,”“e.g.,” “for instance,” “such as,” and “like,” and the verbs“comprising,” “having,” “including,” and their other verb forms, whenused in conjunction with a listing of one or more components or otheritems, are each to be construed as open-ended, meaning that that thelisting is not to be considered as excluding other, additionalcomponents or items. Other terms are to be construed using theirbroadest reasonable meaning unless they are used in a context thatrequires a different interpretation. In addition, the term “and/or” isto be construed as an inclusive or. As an example, the phrase “A, B,and/or C” includes: “A”; “B”; “C”; “A and B”; “A and C”; “B and C”; and“A, B, and C.”

The invention claimed is:
 1. A method of generating a combined rearviewimage of an area behind a vehicle that is towing a trailer, the methodcomprising: capturing a first image from a vehicle camera that isinstalled on the vehicle, the first image being comprised of an areabehind the vehicle; capturing a second image from a trailer camera thatis installed on the trailer, the second image being comprised of an areabehind the trailer; determining an articulation angle ϕ that correspondsto a relative alignment between the vehicle and the trailer, when thevehicle and the trailer are aligned in the same direction thearticulation angle ϕ is zero, when the vehicle and the trailer are notaligned in the same direction the articulation angle ϕ is nonzero;determining an obstructed viewing region within the first image based atleast in part on the articulation angle ϕ, the obstructed viewing regionbeing a region of the first image in which the trailer resides;overlaying at least a portion of the second image at least partiallywithin the obstructed viewing region of the first image, wherein theobstructed viewing region is at least partially based on thearticulation angle ϕ when the at least a portion of the second image isoverlaid; and displaying the combined image on a display within thevehicle; wherein the determining an articulation angle ϕ step furthercomprises determining the articulation angle ϕ by capturing an image ofa trailer frame component with the vehicle camera, using imageprocessing techniques on the image to determine a first reference angleα₁ that corresponds to a relative alignment between the vehicle and thetrailer frame component at that moment, comparing the first referenceangle α₁ to a predetermined reference angle α₀ that corresponds to arelative alignment between the vehicle and the trailer frame componentwhen the vehicle and the trailer are aligned in the same direction, andsubtracting one of the reference angles from the other of the referenceangles to determine the articulation angle.
 2. The method of claim 1,wherein the determining an articulation angle ϕ step further comprisesdetermining the articulation angle ϕ by sensing a steering wheel angleand using the steering wheel angle to determine the articulation angle.3. A method of generating a combined rearview image of an area behind avehicle that is towing a trailer, the method comprising: capturing afirst image from a vehicle camera that is installed on the vehicle, thefirst image being comprised of an area behind the vehicle; capturing asecond image from a trailer camera that is installed on the trailer, thesecond image being comprised of an area behind the trailer; determiningan articulation angle ϕ that corresponds to a relative alignment betweenthe vehicle and the trailer, when the vehicle and the trailer arealigned in the same direction the articulation angle ϕ is zero, when thevehicle and the trailer are not aligned in the same direction thearticulation angle ϕ is nonzero; determining an obstructed viewingregion within the first image based at least in part on the articulationangle ϕ, the obstructed viewing region being a region of the first imagein which the trailer resides; overlaying at least a portion of thesecond image at least partially within the obstructed viewing region ofthe first image, wherein the obstructed viewing region is at leastpartially based on the articulation angle ϕ when the at least a portionof the second image is overlaid; and displaying the combined image on adisplay within the vehicle; wherein the determining an obstructedviewing region step further comprises determining the obstructed viewingregion by retrieving from memory a predefined obstructed viewing regiondefinition that corresponds to an area of the first image that isobstructed by the trailer when the articulation angle ϕ is zero, andmodifying the predefined obstructed viewing region definition based atleast in part on the determined articulation angle ϕ so that themodified obstructed viewing region obstructs different regions of thefirst image than the predefined obstructed viewing region definition. 4.The method of claim 3, wherein the obstructed viewing region is definedby four coordinates that represent vertices of a quadrangle and arelocated within the first image.
 5. The method of claim 1, wherein thedetermining an obstructed viewing region step further comprisesdetermining the obstructed viewing region based at least in part on alength of the trailer.
 6. The method of claim 1, wherein the methodfurther comprises: determining an image overlay area that corresponds toan area on the first image in which the second image will be overlaidand at least partially overlaps with the obstructed viewing region, andthe overlaying step further comprises overlaying the second image withinthe image overlay area of the first image.
 7. A method of generating acombined rearview image of an area behind a vehicle that is towing atrailer, the method comprising: capturing a first image from a vehiclecamera that is installed on the vehicle, the first image being comprisedof an area behind the vehicle; capturing a second image from a trailercamera that is installed on the trailer, the second image beingcomprised of an area behind the trailer; determining an articulationangle ϕ that corresponds to a relative alignment between the vehicle andthe trailer, when the vehicle and the trailer are aligned in the samedirection the articulation angle ϕ is zero, when the vehicle and thetrailer are not aligned in the same direction the articulation angle ϕis nonzero; determining an obstructed viewing region within the firstimage based at least in part on the articulation angle ϕ, the obstructedviewing region being a region of the first image in which the trailerresides; overlaying at least a portion of the second image at leastpartially within the obstructed viewing region of the first image,wherein the obstructed viewing region is at least partially based on thearticulation angle ϕ when the at least a portion of the second image isoverlaid; and displaying the combined image on a display within thevehicle; wherein the method further comprises: determining an imageoverlay area that corresponds to an area on the first image in which thesecond image will be overlaid and at least partially overlaps with theobstructed viewing region, and the overlaying step further comprisesoverlaying the second image within the image overlay area of the firstimage; wherein the method further comprises: identifying a portion ofthe first image that corresponds to a patch region that is within theobstructed viewing region and outside the image overlay area; andrendering graphics in the patch region within the obstructed viewingregion and outside the image overlay area, wherein the graphics areconfigured using at least one of scene memorization and image stitchingto fill the patch region.
 8. The method of claim 3, wherein thedetermining an articulation angle ϕ step further comprises determiningthe articulation angle ϕ by sensing a steering wheel angle and using thesteering wheel angle to determine the articulation angle.
 9. The methodof claim 3, wherein the determining an obstructed viewing region stepfurther comprises determining the obstructed viewing region based atleast in part on a length of the trailer.
 10. The method of claim 3,wherein the method further comprises: determining an image overlay areathat corresponds to an area on the first image in which the second imagewill be overlaid and at least partially overlaps with the obstructedviewing region, and the overlaying step further comprises overlaying thesecond image within the image overlay area of the first image.